Goto

Collaborating Authors

 professional developer


Does In-IDE Calibration of Large Language Models work at Scale?

Koohestani, Roham, Sergeyuk, Agnia, Gros, David, Spiess, Claudio, Titov, Sergey, Devanbu, Prem, Izadi, Maliheh

arXiv.org Artificial Intelligence

The introduction of large language models into integrated development environments (IDEs) is revolutionizing software engineering, yet it poses challenges to the usefulness and reliability of Artificial Intelligence-generated code. Post-hoc calibration of internal model confidences aims to align probabilities with an acceptability measure. Prior work suggests calibration can improve alignment, but at-scale evidence is limited. In this work, we investigate the feasibility of applying calibration of code models to an in-IDE context. We study two aspects of the problem: (1) the technical method for implementing confidence calibration and improving the reliability of code generation models, and (2) the human-centered design principles for effectively communicating reliability signal to developers. First, we develop a scalable and flexible calibration framework which can be used to obtain calibration weights for open-source models using any dataset, and evaluate whether calibrators improve the alignment between model confidence and developer acceptance behavior. Through a large-scale analysis of over 24 million real-world developer interactions across multiple programming languages, we find that a general, post-hoc calibration model based on Platt-scaling does not, on average, improve the reliability of model confidence signals. We also find that while dynamically personalizing calibration to individual users can be effective, its effectiveness is highly dependent on the volume of user interaction data. Second, we conduct a multi-phase design study with 3 expert designers and 153 professional developers, combining scenario-based design, semi-structured interviews, and survey validation, revealing a clear preference for presenting reliability signals via non-numerical, color-coded indicators within the in-editor code generation workflow.


Microsoft's Satya Nadella Doesn't Think Now Is the Time to Stop on AI

TIME - Tech

The last year has been characterized by a rush of new artificial intelligence (AI) programs being released into the world since OpenAI, a lab backed by Microsoft, launched ChatGPT in November 2022. Both Microsoft and Google rolled out products in March that they say will use AI to transform work, and IBM's CEO Arvind Krishna said the company's AI tool will be able to reduce 30 to 50% of repetitive office work. Since taking the helm at Microsoft in 2014, at a time when its market dominance with traditional software offerings was waning, Satya Nadella has focused on ensuring the company remains relevant. . The company has invested heavily in Azure, its cloud computing platform, and in AI, pouring at least $13 billion in the leading lab OpenAI. Microsoft's share price has risen nearly tenfold since Nadella became CEO, outperforming the S&P 500, which has merely doubled its value over the same time.


How Microsoft plans to improve the low-code landscape

#artificialintelligence

Taking on the challenges head-on that stand in the way of their low-code platforms growing, Microsoft's series of new product announcements this week at Build 2022 gives organizations new options for achieving low-code development goals. Microsoft's series of low-code announcements made this week include Power Pages, the latest Microsoft Power Platform addition for creating integrated, scalable and secure websites. Lured by the promises of democratizing app development with visual, declarative, drag and drop interfaces often bundled with enterprise-wide platforms like Microsoft, Salesforce, ServiceNow and others, enterprises have been quick to jump in and experiment. They're learning that support for a low-code platform can get expensive fast once app development moves from small department coding projects to larger-scale, enterprise-wide apps. Low-code platforms' hidden costs include limited process workflow support that further adds to the challenge of scaling them enterprise-wide.


Microsoft named a Leader in The Forrester Wave: Enterprise iPaaS, 2021

#artificialintelligence

Azure Integration Services consisting of Logic Apps, API Management, Event Grid, and Service Bus helps customers connect applications, data, and services on-premises and in the cloud, helping enterprises create new revenue opportunities with an API-driven partner and developer ecosystem and boost productivity with secure and automated workflows. Our vision is to empower all kinds of organizations and users, from citizens to professional developers, to use Azure Integration Services to enable a composable enterprise. We believe that integration is an essential part of modern applications and should be available and accessible to all kinds of users; citizens to professional developers. We reach out to developers at the platform of their choice with Visual Studio and Visual Studio Code extensions, support for Azure DevOps, GitHub Actions, Application Insights, and Azure Monitor, offering an intuitive user experience and increasing developer productivity. With Power Automate and its Windows-integrated robotic process automation (RPA) capabilities, we enable citizen developers to leverage integration without writing code.


Stack Overflow Survey Data Science Highlights - KDnuggets

#artificialintelligence

Every year, Stack Overflow conducts a survey of its users to help inform the development of its community and platform. This year, more than 80,000 developers shared how they learn, the tools and languages they use, and provided all sorts of feedback valuable to Stack Overflow's direction. The results also present a snapshot of developers and development as of when the survey was conducted. The results of the 2021 Stack Overflow Survey were recently shared publicly, along with commentary and insight provided by Stack Overflow. We will take a look at some of the more interesting data points as they pertain to data science, data scientists, and all of the many data-related positions and those professionals who fill them.


Top 11 Process Automation Terms

#artificialintelligence

As automation, low-code, and artificial intelligence grow in terms of adoption the terminology, industry jargon, and acronyms are increasing at a rapid pace. Our team of process automation experts has compiled a list of the Top 11 automation terms to help you go from an automation novice to the subject matter expert in your company. In its pure sense, artificial intelligence (AI) refers to systems which are self-aware, and capable of rational thought. However, in recent years, the term has been used more broadly to encapsulate the simulation of human intelligence processes by machines, especially IT systems. These processes include learning (the acquisition of information and rules for using that information), reasoning (using the rules to reach approximate or definite conclusions), and self-correction (identifying that a course of action is proving or likely to prove unsuccessful and modifying that course).


Microsoft AI Builder brings machine learning to PowerApps

#artificialintelligence

Microsoft's AI Builder artificial intelligence platform, now in preview, enables nonprogrammers -- as well as professional developers -- to easily add AI to the projects they are working on to create more intelligent applications. Microsoft's low-code, no-code Power Platform consists of PowerApps, Power BI and Flow. PowerApps enables developers to create mobile and web apps with low- or no-code. Power BI is for analyzing data, creating reports and creating dashboards with low or no code, and Flow helps devs automate tasks and workflows with low or no code. AI Builder is tightly integrated with PowerApps so that users can simply click on visual prompts to add AI-enabled controls to their mobile or web.


AI Projects No Longer Require a Professional Developer's Touch

#artificialintelligence

What comes to mind when you hear the words "artificial intelligence?" Most of us picture a NASA Skylab space station or a legion of rogue robots from Isaac Asimov's stories. We rarely stop to realize how common AI is and how it has touched our lives. Yet, many people are working on their own AI projects at home. One of the most inspiring stories comes from Nick D'Aloisio, the 15-year-old founder of Trimmit.